Method and system for scalable performance testing in cloud computing environments

ABSTRACT

Certain embodiments of the present disclosure provide techniques for performing performance tests against services in a computing environment. The method generally includes deploying application code to an application namespace hosted on a first set of resources in the computing environment. Testing code is deployed to an infrastructure namespace hosted on a second set of resources in the computing environment. A request to test the application code is received. The request generally includes information identifying a load to be generated in testing the application code. A plurality of container instances implementing the test code are instantiated based on the identified load to be generated to test the application code. A test is executed against the application code through the instantiated plurality of container instances.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 17/187,087, filed Feb. 26, 2021, which is incorporated hereinby reference in its entirety.

INTRODUCTION

Embodiments of the present disclosure relate to testing softwareapplications, and more specifically to scalably performing performancetesting against software applications in cloud computing environments.

BACKGROUND

In many cases, a software application may be deployed in cloud computingenvironments to expose functionality to users of the softwareapplication without needing to install a local copy of the softwareapplication. To serve users of the software application, instances ofthe software application may be deployed in one or more virtual computeinstances (e.g., containers, virtual machines, etc.), with the number ofvirtual compute instances on which the software application is deployedbeing based on the amount of transactions to be processed over a giventime period. For example, one deployment may include instantiating abase number of instances of the software application corresponding to abase load for the application. As the load to be served increases (e.g.,at times of peak demand), additional virtual compute instances can begenerated to serve additional users of the software application.

To ensure that the software application is scalable and performs asexpected, various performance tests can be executed against the softwareapplication in a testing environment. The testing environment may besimilar to the production environment, in that instances of the softwareapplication can be generated and executed in one or more virtual computeinstances in the testing environment prior to deployment to a productionenvironment. However, because each application that may be tested in atesting environment (or different services in an application) may havedifferent performance profiles and expected load, testing theseapplications (or individual services in the applications) may be acomplicated task. For example, establishing a performance testinginfrastructure that generates a small amount of traffic (e.g., a smallnumber of transactions per second) may be a different task fromestablishing a performance testing infrastructure that generates a largeamount of traffic (e.g., a large number of transactions per second).Performance infrastructure that generates a small amount of traffic may,for example, be implemented in a virtual compute instance with fewerresources than performance infrastructure that generates a larger amountof traffic. For example, the virtual compute instance for performanceinfrastructure may be implemented using a smaller number of processorsassigned to the virtual compute instance, slower processors, lessmemory, etc. than would be used for performance infrastructure thatgenerates a larger amount of traffic. Further, differing numbers ofvirtual compute instances may need to be instantiated in order toaccurately test a software application or portions thereof.

Accordingly, techniques are needed to efficiently generate performanceinfrastructure for testing and deploying software applications in cloudcomputing environments.

BRIEF SUMMARY

Certain embodiments provide a computer-implemented method for performingperformance tests against services in a computing environment. Themethod generally includes deploying application code to an applicationnamespace hosted on a first set of resources in the computingenvironment. Testing code is deployed to an infrastructure namespacehosted on a second set of resources in the computing environment. Arequest to test the application code is received. The request generallyincludes information identifying a load to be generated in testing theapplication code. A plurality of container instances implementing thetest code are instantiated based on the identified load to be generatedto test the application code. A test is executed against the applicationcode through the instantiated plurality of container instances.

Still further embodiments provide a computer-implemented method fordeploying application code to a production environment based onperformance tests performed against services in a computing environment.The method generally includes deploying application code to anapplication namespace hosted on a first set of resources in thecomputing environment. Testing code is deployed to an infrastructurenamespace hosted on a second set of resources in the computingenvironment. A request to test the application code is received. Therequest generally includes information identifying a load to begenerated in testing the application code. A plurality of containerinstances implementing the test code are instantiated based on theidentified load to be generated to test the application code. A test isexecuted against the application code through the instantiated pluralityof container instances. A consolidated test result is generated bymerging test results generated by each of the plurality of containerinstances. One or more actions are taken with respect to the applicationcode based on a comparison of the merged test results to a thresholdperformance level for the application code.

Other embodiments provide processing systems configured to perform theaforementioned methods as well as those described herein;non-transitory, computer-readable media comprising instructions that,when executed by one or more processors of a processing system, causethe processing system to perform the aforementioned methods as well asthose described herein; a computer program product embodied on acomputer readable storage medium comprising code for performing theaforementioned methods as well as those further described herein; and aprocessing system comprising means for performing the aforementionedmethods as well as those further described herein.

The following description and the related drawings set forth in detailcertain illustrative features of one or more embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The appended figures depict certain embodiments of the one or moreembodiments and are therefore not to be considered limiting of the scopeof this disclosure.

FIG. 1 illustrates an example computing environment in which testingcode is scalably deployed in a distributed computing environment to testapplication code deployed in an application namespace in the distributedcomputing environment.

FIG. 2 illustrates example operations for performing scalableperformance tests against services in a distributed computingenvironment.

FIG. 3 illustrates example operations for committing test results of ascalable performance test performed against a service in a computingenvironment to a test result repository.

FIG. 4 illustrates example operations for deploying application code toa production environment based on the test results of a scalableperformance test performed against a service in a computing environment.

FIG. 5 illustrates an example computing system on which embodiments ofthe present disclosure may be implemented.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe drawings. It is contemplated that elements and features of oneembodiment may be beneficially incorporated in other embodiments withoutfurther recitation.

DETAILED DESCRIPTION

Embodiments of the present disclosure provide apparatuses, methods,processing systems, and computer-readable mediums for scalablyperforming performance tests against application code deployed in adistributed computing environment. Generally, application code may bedeployed to one or more virtual compute instances in a distributedcomputing environment, and one or more container virtual computeinstances may be instantiated in the distributed computing environmentto execute various tests against the application code. The properties ofthe containerized virtual compute instances in which test code executesmay be based on the properties of the application code to be tested,such as an expected amount of traffic that the application code isexpected to process over a given time period, so that application codeis tested using the appropriate amount of resources. Application codeexpected to experience high loading in real life (e.g., in a productionenvironment) may be tested using test code deployed to virtual computeinstances capable of generating a large amount of traffic. In contrast,application code expected to experience lower loading in real life maybe tested using test code deployed to virtual compute instances capableof generating a smaller amount of traffic but incapable of generatingthe higher amount of traffic so that such application code is not testedfor an unrealistically high amount of traffic. Generally, these virtualcompute instances may have differing amounts of compute resources (e.g.,processors, processor power, memory, etc.) with correspondinglydifferent maximum amounts of traffic that can be generated. Thus, avirtual compute instance with a large amount of compute resources mayhave a larger maximum activity generation capability than a virtualcompute instance with a smaller amount of compute resources.

Performance testing is an integral part of the software developmentprocess. Because software applications are generally expected to performat a certain level, performance testing is generally used in thesoftware development process to test application code prior todeployment in a production environment. Generally, application code thatdelivers the expected performance may be deployed to a productionenvironment, while application code that does not deliver the expectedperformance may be blocked from deployment until the performance issuesidentified during performance testing are rectified. To test applicationcode and ensure that the application code provides the expected level ofperformance prior to deployment, test code can be written to generatetraffic that the application code is expected to process. If theapplication code consistently fails to provide a response to the trafficgenerated by the test code within a defined amount of time or providesincorrect responses, the application code may be deemed defective andblocked from deployment to a production environment. Otherwise, theapplication code may be deemed to comply with the expected behavior andmarked as ready for deployment to the production environment.

Some software applications may be implemented as a collection ofservices or microservices. In such an implementation, each service mayprovide a specific portion of the functionality of the application. Forexample, one service may provide user authentication functionality;another service may allow for the execution of a specific workflow orportion thereof; still another service may provide for the processing ofsupport requests; and so on. Each of the services in the application mayhave different performance characteristics. Testing these services usinga same test code configuration may be inappropriate, as some servicesmay be tested using an amount of traffic in excess of the expectedamount of traffic, while other services may be tested using an amount oftraffic that is significantly less than the expected amount of traffic.Thus, some services may be erroneously deemed as nonperformant, whichmay delay the deployment of application code. Further, other servicesmay be erroneously deemed as compliant with performance requirements,which may allow code that ultimately degrades the performance of theapplication to be deployed to a production environment.

Embodiments described herein generally provide a scalable testinginfrastructure for testing application code in a distributed computingenvironment. Generally, a scalable testing infrastructure may allow forthe dynamic allocation and generation of virtual compute instances fortesting application code in a development environment so that the numberof virtual compute instances and the compute resource configuration foreach of the virtual compute instances is dynamically generated for theapplication code being tested. By dynamically generating test codecontainer instances that are tailored to the performance requirements orspecifications for a specific application or portion thereof (e.g., aspecific service exposing functionality within an application),application code may be tested using an expected amount of traffic andan appropriate amount of compute resources for testing the applicationcode using the expected amount of traffic. Thus, application code may betested using realistic testing scenarios, and deployment of applicationcode that has been tested against realistic loads to a productionenvironment.

By testing application code using a scalable testing infrastructure,aspects of the present disclosure may minimize the amount of computeresources used in testing application code. Within a distributed,on-demand computing environment, resources not needed for testingapplication code for the amount of traffic expected to be processed in aproduction environment may remain free for use by other applications orservices in the distributed computing environment. Thus, testingapplication code using a scalable testing infrastructure may improve theaccuracy of testing scenarios and allow for performant code to bedeployed to a production environment, which may improve the reliabilityand availability of applications deployed to a production environment,as application code deployed to the production environment may beverified to work as expected for the expected amount of traffic expectedto be processed by the application code. Further, because infrastructurefor testing application code may be dynamically generated, aspects ofthe present disclosure may optimize the amount of computing resourcesneeded for testing so that excess resources that would not be needed arenot allocated for testing purposes. This may increase the amount ofcompute resources available in a distributed computing environment, andthese available compute resources may be used to support existingapplications experiencing additional load so that these existingapplications continue to perform as expected or to new applicationsdeployed within the distributed computing environment.

Example Systems for Scalable Performance Testing of Application Code ina Distributed Computing Environment

FIG. 1 illustrates an example computing environment 100 in which testcode is scalably deployed in a distributed computing environment fortesting application code based on the expected performancecharacteristics of the application code.

As illustrated, computing environment 100 includes an application sourcecode repository 112, infrastructure source code repository 114, testingsource code repository 116, project build system 120, distributedcompute environment 130, test result repository 140, and performanceanalyzer 150.

To test application code, project build system 120 uses code from eachof application source code repository 112, infrastructure source coderepository 114, and testing source code repository 116 to compile one ormore instances of the application code and the testing code and deploysthe compiled instances to distributed compute environment 130 forexecution. Generally, to deploy application code for testing, projectbuild system 120 retrieves the application code from source coderepository and compiles the application code into one or more executablefiles. Project build system 120 can then instantiate an applicationnamespace 132 in distributed compute environment 130 in which one ormore executable files execute. Generally, the application namespace 132represents a set of virtual compute instances accessible using the samename by test code executing in an infrastructure namespace 136 to testthe application code. Project build system 120 can then deploy thecompiled application code into one or more application instances 134,which represent discrete virtual compute instances on which the compiledapplication code executes.

In some aspects, the number of application instances 134 instantiated byproject build system 120 may be commensurate with a number of virtualcompute instances on which the compiled application code would bedeployed in a production environment. Project build system 120 caninstantiate the application instances 134 based on a configuration ofthe virtual machines on which the compiled application code would bedeployed in a production environment so that the compiled applicationcode is tested on virtual compute instances with the same or similarcomputing resources (e.g., processor power, memory, memory latency,etc.) in a testing environment.

For example, the configuration information associated with theapplication source code may explicitly define the configuration (e.g.,number of processors or processing cores, types of processors orprocessing cores, amount of system memory, etc.) of the virtual machineson which the compiled application code is deployed in a productionenvironment, and project build system 120 can instantiate virtualcompute instances for application instances 134 with the sameconfiguration.

In another example, the configuration information associated with theapplication source code may identify a baseline performance level of thevirtual compute instances on which the compiled application code isdeployed in a production environment (e.g., a number of floating pointoperations per second (FLOPS) that the virtual compute instances in theproduction environment are capable of performing). Project build system120 can request that a deployment manager in distributed computeenvironment 130 generate virtual compute instances for applicationinstances 134 according to the configuration information, which mayguarantee that the application code is being tested on the same orsimilar system configurations and that the recorded performance of thecompiled application code in application instances 134 is similar tothat which would be experienced in a production environment.

To test the application source code in application namespace 132,project build system 120 can use the infrastructure source code ininfrastructure source code repository 114 to generate and instantiateinfrastructure namespace 136. Infrastructure namespace 136 generallyrepresents a set of virtual compute instances accessible using the samename that is used to test application source code in applicationnamespace 132. Generally, the infrastructure source code ininfrastructure source code repository 114 includes code for establishinga namespace associated with the container instances 138 such that any ofthe container instances 138 in infrastructure namespace 136 can generateand send transaction requests to application namespace 132 forexecution. After project build system 120 builds and deploysinfrastructure namespace from the infrastructure source code ininfrastructure source code repository 114, the test code can be compiledand deployed in one or more container instances 138 in infrastructurenamespace 136.

To generate the container instances 138 for testing the compiledapplication source code executing in application instances 134, projectbuild system 120 can use information about an expected number oftransactions to be executed over a given time period the applicationsource code and a number of transactions that a virtual compute instancecan generate in order to determine the number and configuration ofvirtual compute instances to generate for container instances 138. Insome aspects, project build system 120 can select the number andconfiguration of the virtual compute instances for container instances138 according to a resource minimization strategy.

For example, if distributed compute environment 130 allow for thegeneration of virtual compute instances that support X transactions persecond (TPS), Y TPS, and Z TPS, where X=3*Y and Y=3*Z, and theapplication source code is expected to process 2Y+Z TPS, project buildsystem 120 can determine that the test can be performed using onevirtual compute instance supporting X TPS or two virtual computeinstances supporting Y TPS and one virtual compute instance supporting ZTPS. A resource minimization strategy that is configured to minimize thenumber of virtual compute instances generated within infrastructurenamespace 136 may thus result in project build system 120 instantiatinga single virtual compute instance supporting X TPS for the test. Aresource minimization strategy that is configured, however, to minimizea total cost of performing a test may result in project build system 120instantiating two virtual compute instances supporting Y TPS and onevirtual compute instance supporting Z TPS, if the total cost is lessthan the total cost of executing the test using the single virtualcompute instance supporting X TPS.

In some aspects, a plurality of virtual compute instances may beorganized into a resource pod in distributed compute environment 130.Each resource pod may be associated with a maximum load (e.g., a maximumnumber of transaction requests per second) that can be generated againstapplication instances 134 in application namespace 132. A number ofresource pods may be instantiated by project build system 120 to executethe test code based on the maximum load associated with each resourcepod available in distributed compute environment 130 and an identifiedload to be generated in testing the application source code executing inapplication instances 134. In some aspects, the maximum load may bedefined in terms of a number of read requests and a number of writerequests to be executed against virtual compute instances in theapplication namespace 132 over a given period of time (e.g., a number ofread requests to process per second and a number of write requests toprocess per second).

Generally, project build system 120 may allow for any number of virtualcompute instances in distributed compute environment 130 to beinstantiated in order to test compiled application code executing onapplication instances 134 in application namespace 132. Because thenumber of container instances 138 and the configuration of the containerinstances 138 generated by project build system 120 may be dynamicallydefined according to the traffic profile of the application source codebeing tested, an appropriate amount of compute resources may beallocated for testing the application source code. Excess resources thatmay not be needed for testing application source code within distributedcompute environment 130 may thus be allocated for other uses, such asconcurrent testing of other application source code in a differentapplication namespace. Because only the appropriate amount of computeresources are allocated for testing the application source code, aspectsof the present disclosure may improve resource availability within thedistributed compute environment 130, which may allow for additionalcompute resources to be used to improve or maintain the performance ofother applications executing within distributed compute environment 130or allow for additional applications to be tested and deployed withindistributed compute environment 130.

After the virtual compute instances for container instances 138 areinstantiated, project build system 120 can deploy the compiled test codeto container instances 138 for execution. In some aspects, the same testcode may be deployed to container instances 138; however, in someaspects, different test code may be deployed to container instances 138based on the compute resources allocated to each of the individualcontainer instances. Once the compiled test code is deployed to thecontainer instances 138, testing may commence against the applicationsource code deployed in application instances 134. As tests are executedagainst application source code deployed in application instances 134,the test code executing in container instances 138 can collectperformance data for the test. The performance data may include, forexample, timing information tracking the amount of time elapsed betweentransmission of a transaction request to application namespace 132 andreceipt of a response from one of the application instances 134 inapplication namespace 132. In some aspects, the test code may include athreshold or benchmark time between transmission of a transactionrequest and receipt of a response from an application instance 134. Ifthe measured time is less than the threshold or benchmark time, the testcode executing in container instances 138 can mark the transactionrequest as having been successfully processed; otherwise, the test codecan mark the transaction request as having failed. In some aspects, thetest code deployed in container instances 138 may include aspecification of a correct response for each transaction generated andtransmitted to application instances 134 for execution. If the responsefrom one of the application instances 134 matches the specified correctresponse, the test code executing in container instances 138 can markthe response as correct and, thus, that the transaction request wassuccessfully processed. Otherwise, the test code executing can mark theresponse as incorrect, and thus, can determine that the applicationsource code failed to process the transaction request.

A test executed within infrastructure namespace 136 may have a definedtermination condition (e.g., based on a total number of transactionrequests generated, a total amount of time, a threshold number of failedresponses, etc.). When the defined termination condition is satisfied,the test code executing in container instances 138 may terminate testingand generate a report summarizing the results of the tests executedagainst compiled application source code in application namespace 132.The report generated by the test code in any given container instance138 may include information identifying a number of transaction requeststransmitted to application namespace 132, information about the numberor proportion of transaction requests that were successfully executed,timing information, correctness information, and the like. In someaspects, the report may further include information about each of thetransaction requests transmitted to application namespace 132, whichdevelopers of the application source can use in debugging theapplication source code. The report may be committed to test resultrepository 140, which represents a data repository from whichperformance analyzer 150 can pull data in order to determine whether theapplication source code being tested is ready for deployment in aproduction environment and from which developers can access testinginformation for use in debugging and developing the application sourcecode.

In some aspects, one of the one or more container instances 138 may bedesignated as a lead container instance. The lead container instance maytrack the total number of transaction requests generated in testingapplication source code and may issue a termination command to the othercontainer instances 138 when a test termination condition is satisfied.

In some aspects, the test code in the lead container instance may beconfigured to generate a consolidated test result based on the testresults generated by the test code executing in each of the plurality ofcontainer instances. To generate the consolidated test result, the testcode in the lead container instance can retrieve the test resultsgenerated by each of the plurality of container instances 138. Theretrieved test results may be consolidated into a single test result bymerging (e.g., adding, averaging, etc.) the retrieved test results. Forexample, a consolidated test result may include the total number oftransaction requests generated by the container instances 138, anaverage response time over all of the transaction requests, a totalnumber or percentage of successfully processed transaction requestsacross all of the container instances 138, and so on. The consolidatedtest result may then be committed to test result repository 140 forfurther use, such as performance trend analysis, determining whether topromote application source code from a development environment to aproduction environment, and so on.

Performance analyzer 150 generally uses the results of the testsexecuted by test code deployed to container instances 138 to determinewhether the application source code under test meets the definedperformance requirements for the application source code. The definedperformance requirements for the application source code may be defined,for example, as a threshold average response time, threshold responsecorrectness level, or the like.

If the test results indicate that the application performs as expected,performance analyzer 150 can deploy the application source code fromapplication source code repository 112 to a production environmenthosted in distributed compute environment 130 (not shown). To deploy theapplication source code, performance analyzer 150 can, in some aspects,instruct project build system 120 to compile the application source codeand deploy the compiled application source code to one or more virtualcompute instances associated with the production environment indistributed compute environment 130.

In some aspects, project build system 120 can be used to perform othertypes of testing against application code executing in applicationnamespace 132. For example, project build system 120 may be configuredto perform chaos testing against application code in applicationnamespace 132. Generally, chaos testing allows for various failurescenarios to be performed in order to test the resiliency of applicationcode to unexpected scenarios, such as simulated traffic overloadscenarios, unexpected system failure, or the like. In some aspects,chaos testing may be used to simulate a cascading failure of multiple,interrelated application components. To execute a chaos test againstapplication code, thus, project build system 120 may be configured toinitially generate virtual compute instances in the applicationnamespace 132 and infrastructure namespace 136 corresponding to atypical real-world load on the application code under test.Subsequently, project build system 120 may simulate various failure orotherwise chaotic scenarios against the application code executing inthe application namespace 132. For example, to simulate a suddenincrease in traffic, project build system 120 can instantiate additionalcontainer instances 138 in which test code can execute and generateadditional traffic to be processed by application instances 134 inapplication namespace 132. In another example, to simulate a failure ofone or more application components (e.g., services, microservices,etc.), project build system 120 can terminate one or more applicationinstances 134 in application namespace 132. Other failures can also oralternatively be injected into application namespace 132 to execute achaos test against application code. Chaos testing using chaos workflowsorchestrated by project build system 120 may allow for automation andself-service of chaos testing, which may allow for efficient executionof chaos tests in a development environment while avoiding or minimizingnegative impacts on a production system.

Example Execution of Scalable Performance Tests Against Services in aDistributed Computing Environment

FIG. 2 illustrates example operations 200 for scalably performingperformance tests against services in a distributed computingenvironment. The operations described herein may be performed by one ormore systems in a computing environment, such as project build system120 and distributed compute environment 130 illustrated in FIG. 1 .

As illustrated, operations 200 begin at block 210, where a systemdeploys application code to an application namespace hosted on a firstset of resources in the computing environment. Generally, to deploy theapplication code to the application namespace, the system caninstantiate an application namespace and a plurality of virtual computeinstances on which the application code (or a compiled version of theapplication code) is to execute. The application namespace may representa common name that test code executing on one or more virtual computeinstances in an infrastructure namespace can reference to sendtransaction requests to the application code for processing. Generally,the resources on which the application code may be deployed may includeone or more virtual compute instances available in a distributedcomputing environment. These virtual compute instances may beprovisioned similarly to the virtual compute instances on which theapplication code (or compiled version of the application code) executesin a production environment, so that the application is tested in anenvironment that is the same as or similar to the productionenvironment.

At block 220, the system deploys testing code to an infrastructurenamespace hosted on a second set of resources in the computingenvironment. To deploy the testing code, the system can generate aninfrastructure namespace representing a common name that applicationcode executing in the application namespace can reference intransmitting responses to received transaction requests. Generally, theresources on which the testing code may be deployed may include aplurality of virtual compute instances. Each of the virtual computeinstances may expose different processing capabilities so that a mix ofvirtual compute instances can be selected and instantiated to test theapplication source code according to the expected number of transactionsthe application source code is to process over a given time period.

In some aspects, the system can generate the infrastructure namespace bycompiling infrastructure code and deploying the compiled infrastructurecode. The infrastructure code may expose functionality for creating,maintaining, and terminating container instances (or groups of containerinstances) in which the test code executes. In some aspects, theinfrastructure code may also expose functionality for terminating theinfrastructure namespace after execution of a test is completed.

At block 230, the system receives a request to test the applicationcode. The request generally includes information identifying a load tobe generating in testing the application code. The identified load mayinclude, for example, a number of transaction requests to be processedby the application code over a given amount of time (e.g., per second).The transaction requests may, in some aspects, be further broken downinto read requests and write requests, as the number of read and writerequests over a given time period may differ and read and write requestsmay have different performance profiles (e.g., read requests maygenerally be executed faster than write requests).

At block 240, the system instantiates a plurality of container instancesimplementing the test code based on the identified load to be generatedto test the application code. To determine the number of containerinstances and the configuration of the container instances to beinstantiated to test the application code, the system can useinformation about an amount of traffic that each type of virtual computeinstance in the distributed computing environment can generate and thetotal amount of traffic over a given time period to be generated againstthe application code deployed to the application namespace. A minimalset of container instances may be instantiated to satisfy the totalamount of traffic to be generated against the application code deployedto the application namespace. In some aspects, the minimal set ofcontainer instances may be the smallest number of virtual computeinstances that can be used to generate the total amount of traffic. Inanother aspect, the minimal set of container instances may be the numberof virtual compute instances that result in the generation of an amountof traffic that is closest to the specified total amount of traffic. Instill further aspects, the minimal set of container instances may be thevirtual compute instances that generate the total amount of traffic atthe lowest cost.

At block 250, the system executes a test against the application codethrough the instantiated plurality of container instances.

In some aspects, the test may be terminated when a terminating conditionis met. For example, the termination condition may include a durationover which the test is to be executed. When the system determines thatan elapsed time over which the test has executed equals or exceeds theidentified duration, the system can terminate the test. To terminate thetest, the system can terminate each of the plurality of containerinstances implementing the test code. In some aspects, in terminating acontainer instance, the system can instruct the container instance todiscontinue sending transaction requests to application code deployed inthe application namespace and to commit the final test results to a testresult repository. The final test results may be committed to the testresult repository when a response to the last transaction request issuedby the test code to the application code deployed in the applicationnamespace is received and processed. Once the final test results arecommitted to the test result repository, the container instancesimplementing the test code can be terminated, and the resourcesassociated with the terminated container instances can be redeployedwithin a distributed computing environment for other workloads.

Example Generation of Consolidated Test Results of Performance TestsAgainst Executed Services in a Distributed Computing Environment

FIG. 3 illustrates example operations 300 that may be performed togenerate a consolidated test result from the plurality of containerinstances implementing the test code.

As illustrated, operations 300 may continue at block 310 from block 250illustrated in FIG. 2 . At block 310, the system retrieves test resultsgenerated by each of the plurality of container instances implementingthe test code. The test results generated by a specific containerinstance may include, for example, a summary test result specifying atotal number of tests executed against the application code, averageresponsiveness data, average correctness data, and/or other aggregatedmetrics for the tests executed by the specific container instance. Inanother aspect, the test results may include information associated witheach discrete test executed against the application code.

At block 320, the system generates a consolidated test result. Togenerate the consolidated test result, the system can merge the testresults generated by each of the container instances into a single testresult representing the recorded performance of the application codeacross the test code implemented in each of the container instances. Theconsolidated test result may thus include information about the totalnumber of tests executed against the application code, the total numberof tests that were executed successfully by the application code, thepercentage of successfully executed tests, and so on.

At block 330, the system commits the consolidated test result to a testreport repository. The consolidated test result may be used, forexample, to identify application code that requires further developmentto achieve performance objectives for the code and to determine whetherapplication code is in condition for deployment into a productionenvironment.

Example Deployment of Application Code Based on Scalable PerformanceTests Executed Against Services in a Distributed Computing Environment

FIG. 4 illustrates example operations 400 may be performed to deployapplication code to a production environment in a continuous integrationand continuous development (Cl/CD) platform based on the consolidatedresults of tests executed against application code in an applicationnamespace. Generally, in a Cl/CD platform, performance tests may beperformed as application code changes, and each change is deployed froma development environment to a production environment as the performanceand functionality of the application code is verified through theexecution of one or more tests.

As illustrated, operations 400 may continue at block 410 from block 250illustrated in FIG. 2 . At block 410, the system retrieves test resultsgenerated by each of the plurality of container instances implementingthe test code. The test results generated by a specific containerinstance may include, for example, a summary test result specifying atotal number of tests executed against the application code, averageresponsiveness data, average correctness data, and/or other aggregatedmetrics for the tests executed by the specific container instance. Inanother aspect, the test results may include information associated witheach discrete test executed against the application code.

At block 420, the system generates a consolidated test result. Togenerate the consolidated test result, the system can merge the testresults generated by each of the container instances into a single testresult representing the recorded performance of the application codeacross the test code implemented in each of the container instances. Theconsolidated test result may thus include information about the totalnumber of tests executed against the application code, the total numberof tests that were executed successfully by the application code, thepercentage of successfully executed tests, and so on.

At block 430, the system deploys the application code to a productionenvironment based on determining that the consolidated test results meeta threshold performance level established for the application code. Indeploying the application code to a production environment, the systemcan compile the application code and replace a previous version of theapplication code executing on virtual compute instances associated withthe production environment with the newly compiled version of theapplication code. In some aspects, the system can terminate the virtualcompute instances on which a previous version of the application code isexecuting and can re-designate the virtual compute instances in theapplication namespace as production environment compute resources.

If, however, the system determines that the consolidated test results donot satisfy the threshold performance level established for theapplication code, the system can place a block on the application sourcecode in application source code repository 112 indicating that thecurrent version of the application source code is not ready fordeployment to a production environment. The system can, in some aspects,generate an alert to one or more developers responsible for developingthe application source code indicating that the application source codefailed a performance test. The system may remove the block on deployingthe application source code if the consolidated test results of asubsequent test indicate that the application code is now in conditionfor deployment to a production environment (e.g., meets the performancerequirements defined for the application code).

Example Systems for Scalable Performance Testing of Application Code ina Distributed Computing Environment

FIG. 5 illustrates an example system 500 that can scalably executeperformance testing of application code in a distributed computingenvironment. For example, system 500 may comprise one more ofapplication source code repository 112, infrastructure source coderepository 114, testing source code repository 116, project build system120, distributed compute environment 130, test result repository 140,and/or performance analyzer 150 illustrated in FIG. 1 .

As shown, system 500 includes a central processing unit (CPU) 502, oneor more I/O device interfaces 504 that may allow for the connection ofvarious I/O devices 514 (e.g., keyboards, displays, mouse devices, peninput, etc.) to the system 500, network interface 506 through whichsystem 500 is connected to network 550 (which may be a local network, anintranet, the internet, or any other group of computing devicescommunicatively connected to each other), a memory 508, storage 510, andan interconnect 512.

CPU 502 may retrieve and execute programming instructions stored in thememory 508. Similarly, the CPU 502 may retrieve and store applicationdata residing in the memory 508. The interconnect 512 transmitsprogramming instructions and application data, among the CPU 502, I/Odevice interface 504, network interface 504, memory 508, and storage510.

CPU 502 is included to be representative of a single CPU, multiple CPUs,a single CPU having multiple processing cores, and the like.

Memory 508 is representative of a volatile memory, such as a randomaccess memory, or a nonvolatile memory, such as nonvolatile randomaccess memory, phase change random access memory, or the like. As shown,memory 508 includes an application source code repository 520,infrastructure source code repository 530, testing source coderepository 540, project build system 550, distributed computeenvironment 560, test result repository 570, and performance analyzer580.

Application source code repository 520, infrastructure source coderepository 530, and testing source code repository 540 may correspond toapplication source code repository 112, infrastructure source coderepository 114, and testing source code repository 116, respectively,illustrated in FIG. 1 . Each of these repositories generally representdata stores in which source code compiled by project build system 550 isstored. Source code for an application (or portion thereof) being testedmay be stored in application source code repository 520. Source code forgenerating an infrastructure namespace in which virtual computeinstances hosting compiled test code may be stored in infrastructuresource code repository 114. Finally, source code for executing testsagainst an application may be stored in testing source code repository116.

Project build system 550 may correspond to project build system 120illustrated in FIG. 1 . Generally, project build system 550 uses thesource code in application source code repository 520 to build instancesof the application for deployment in one or more virtual computeinstances associated with an application namespace in distributedcompute environment 560. Project build system 550 uses the source codein infrastructure source code repository 530 to build an infrastructurenamespace in distributed compute environment 560 in which containerinstances for the test code executes. Finally, project build system 550uses the source code in testing source code repository 540 to build anddeploy the test code to container instances in the infrastructurenamespace to execute tests against the application hosted in theapplication namespace.

Distributed compute environment 560 may correspond to distributedcompute environment 130 illustrated in FIG. 1 and represent a pluralityof virtual compute instances organized into an application namespace andan infrastructure namespace. Application code compiled by project buildsystem 550 may be deployed to virtual compute instances associated withthe application namespace and may receive and process transactionrequests from test code executing in container instances associated withthe infrastructure namespace. Infrastructure code compiled by projectbuild system 550 may generate the infrastructure namespace, and testcode compiled by project build system 550 may be deployed to containerinstances associated with the infrastructure namespace for generatingtransaction requests and other tests against the application codedeployed to virtual compute instances associated with the applicationnamespace.

Test result repository 570 may correspond to test result repository 140illustrated in FIG. 1 . Generally, test result repository 570 may storethe results of tests executed against the application code hostedvirtual compute instances associated with the application namespace.Test result repository 570 may be a searchable data repository thatallows developers of a software application to search for applicationcomponents that failed testing to identify which application componentson which development should be focused. In some aspects, the data intest result repository 570 may also or alternatively be used as an inputinto a continuous integration/continuous development (Cl/CD) pipeline(e.g., implemented by performance analyzer 580) to determine whetherapplication source code is ready for promotion from a testingenvironment to a production environment.

Performance analyzer 580 may correspond to performance analyzer 150illustrated in FIG. 1 . Generally, performance analyzer 580 uses thetest results stored in test result repository 570 and a thresholdperformance level defined for an application (or component thereof) todetermine whether application source code is in condition for promotionfrom a test environment to a production environment. If the test resultsmeet the threshold performance level defined for the application,performance analyzer 580 can deploy the application source code to aproduction environment and replace a previous version of the applicationsource code with the tested, new version of the application source code.Otherwise, performance analyzer 580 can block deployment of theapplication source code until the results of subsequent tests indicatethat the application source code compiles with the threshold performancelevel for the application (or component thereof).

Example Clauses

Clause 1: A method for performing performance tests against services ina computing environment, comprising: deploying application code to anapplication namespace hosted on a first set of resources in thecomputing environment; deploying testing code to an infrastructurenamespace hosted on a second set of resources in the computingenvironment; receiving a request to test the application code, therequest including information identifying a load to be generated intesting the application code; instantiating a plurality of containerinstances implementing the test code based on the identified load to begenerated to test the application code; and executing a test against theapplication code through the instantiated plurality of containerinstances.

Clause 2: The method of Clause 1, further comprising: retrieving testresults generated by each of the plurality of container instancesimplementing the test code; and generating a consolidated test result bymerging the test results generated by each of the plurality of containerinstances.

Clause 3: The method of Clause 2, further comprising: committing theconsolidated test result to a test report repository.

Clause 4: The method of Clauses 2 or 3, further comprising: deployingthe application code to a production environment based on determiningthat the consolidated test results meet a threshold performance levelestablished for the application code.

Clause 5: The method of any of Clauses 1 through 4, further comprising:compiling infrastructure code to generate the infrastructure namespace,wherein the infrastructure code exposes functionality for creating,maintaining, and terminating groups of container instances in which testcode executes.

Clause 6: The method of any of Clauses 1 through 5, whereininstantiating the plurality of container instances implementing the testcode comprises instantiating a plurality of resource pods on which thetest code is to be executed based on the identified load to be generatedand a maximum load that can be generated by each of a plurality ofresource pod configurations.

Clause 7: The method of Clause 6, wherein each respective configurationof the plurality of resource pod configurations specifies an amount ofmemory and an amount of processing power corresponding to the maximumload the respective configuration is capable of generating against codeexecuting in the application namespace.

Clause 8: The method of any of Clauses 1 through 7, wherein theinformation identifying the load to be generated comprises informationidentifying a number of read requests to be executed against theapplication namespace and a number of write requests to be executedagainst the application namespace over a specified time period.

Clause 9: The method of any of Clauses 1 through 8, wherein theinformation identifying the load to be generated comprises informationidentifying a duration over which the test is to be executed.

Clause 10: The method of Clause 9, further comprising: determining thatan elapsed time over which the test has executed equals the identifiedduration over which the test is to be executed, and terminating theplurality of container instances implementing the test code based ondetermining that the elapsed time equals the identified duration overwhich the test is to be executed.

Clause 11: A method for deploying application code to a productionenvironment based on performance tests performed against services in acomputing environment, comprising: deploying application code to anapplication namespace hosted on a first set of resources in thecomputing environment; deploying testing code to an infrastructurenamespace hosted on a second set of resources in the computingenvironment; receiving a request to test the application code, therequest including information identifying a load to be generated intesting the application code; instantiating a plurality of containerinstances implementing the test code based on the identified load to begenerated to test the application code; executing a test against theapplication code through the instantiated plurality of containerinstances; retrieving test results generated by each of the plurality ofcontainer instances implementing the test code; generating aconsolidated test result by merging the test results generated by eachof the plurality of container instances; and deploying the applicationcode to a production environment based on determining that theconsolidated test results meet a threshold performance level establishedfor the application code.

Clause 12: A system, comprising: a memory having computer-executableinstructions stored thereon; and a processor configured to execute thecomputer-executable instructions to cause the system to perform theoperations of any of Clauses 1 through 11.

Clause 13: A system, comprising: means for performing the operations ofany of Clauses 1 through 11.

Clause 14: A computer-readable medium having instructions stored thereonwhich, when executed by a processor, performs the operations of any ofClauses 1 through 11.

Additional Considerations

The preceding description is provided to enable any person skilled inthe art to practice the various embodiments described herein. Variousmodifications to these embodiments will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other embodiments. For example, changes may be made in thefunction and arrangement of elements discussed without departing fromthe scope of the disclosure. Various examples may omit, substitute, oradd various procedures or components as appropriate. Also, featuresdescribed with respect to some examples may be combined in some otherexamples. For example, an apparatus may be implemented or a method maybe practiced using any number of the embodiments set forth herein. Inaddition, the scope of the disclosure is intended to cover such anapparatus or method that is practiced using other structure,functionality, or structure and functionality in addition to, or otherthan, the various embodiments of the disclosure set forth herein. Itshould be understood that any aspect of the disclosure disclosed hereinmay be embodied by one or more elements of a claim.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: a, b, or c” is intended to cover a, b, c,a-b, a-c, b-c, and a-b-c, as well as any combination with multiples ofthe same element (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b,b-b-b, b-b-c, c-c, and c-c-c or any other ordering of a, b, and c).

As used herein, the term “determining” encompasses a wide variety ofactions. For example, “determining” may include calculating, computing,processing, deriving, investigating, looking up (e.g., looking up in atable, a database or another data structure), ascertaining and the like.Also, “determining” may include receiving (e.g., receiving information),accessing (e.g., accessing data in a memory) and the like. Also,“determining” may include resolving, selecting, choosing, establishingand the like.

The methods disclosed herein comprise one or more steps or actions forachieving the methods. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isspecified, the order and/or use of specific steps and/or actions may bemodified without departing from the scope of the claims. Further, thevarious operations of methods described above may be performed by anysuitable means capable of performing the corresponding functions. Themeans may include various hardware and/or software component(s) and/ormodule(s), including, but not limited to a circuit, an applicationspecific integrated circuit (ASIC), or processor. Generally, where thereare operations illustrated in figures, those operations may havecorresponding counterpart means-plus-function components with similarnumbering.

The various illustrative logical blocks, modules and circuits describedin connection with the present disclosure may be implemented orperformed with a general purpose processor, a digital signal processor(DSP), an application specific integrated circuit (ASIC), a fieldprogrammable gate array (FPGA) or other programmable logic device (PLD),discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A general-purpose processor may be a microprocessor, but in thealternative, the processor may be any commercially available processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

A processing system may be implemented with a bus architecture. The busmay include any number of interconnecting buses and bridges depending onthe specific application of the processing system and the overall designconstraints. The bus may link together various circuits including aprocessor, machine-readable media, and input/output devices, amongothers. A user interface (e.g., keypad, display, mouse, joystick, etc.)may also be connected to the bus. The bus may also link various othercircuits such as timing sources, peripherals, voltage regulators, powermanagement circuits, and the like, which are well known in the art, andtherefore, will not be described any further. The processor may beimplemented with one or more general-purpose and/or special-purposeprocessors. Examples include microprocessors, microcontrollers, DSPprocessors, and other circuitry that can execute software. Those skilledin the art will recognize how best to implement the describedfunctionality for the processing system depending on the particularapplication and the overall design constraints imposed on the overallsystem.

If implemented in software, the functions may be stored or transmittedover as one or more instructions or code on a computer-readable medium.Software shall be construed broadly to mean instructions, data, or anycombination thereof, whether referred to as software, firmware,middleware, microcode, hardware description language, or otherwise.Computer-readable media include both computer storage media andcommunication media, such as any medium that facilitates transfer of acomputer program from one place to another. The processor may beresponsible for managing the bus and general processing, including theexecution of software modules stored on the computer-readable storagemedia. A computer-readable storage medium may be coupled to a processorsuch that the processor can read information from, and write informationto, the storage medium. In the alternative, the storage medium may beintegral to the processor. By way of example, the computer-readablemedia may include a transmission line, a carrier wave modulated by data,and/or a computer readable storage medium with instructions storedthereon separate from the wireless node, all of which may be accessed bythe processor through the bus interface. Alternatively, or in addition,the computer-readable media, or any portion thereof, may be integratedinto the processor, such as the case may be with cache and/or generalregister files. Examples of machine-readable storage media may include,by way of example, RAM (Random Access Memory), flash memory, ROM (ReadOnly Memory), PROM (Programmable Read-Only Memory), EPROM (ErasableProgrammable Read-Only Memory), EEPROM (Electrically ErasableProgrammable Read-Only Memory), registers, magnetic disks, opticaldisks, hard drives, or any other suitable storage medium, or anycombination thereof. The machine-readable media may be embodied in acomputer-program product.

A software module may comprise a single instruction, or manyinstructions, and may be distributed over several different codesegments, among different programs, and across multiple storage media.The computer-readable media may comprise a number of software modules.The software modules include instructions that, when executed by anapparatus such as a processor, cause the processing system to performvarious functions. The software modules may include a transmissionmodule and a receiving module. Each software module may reside in asingle storage device or be distributed across multiple storage devices.By way of example, a software module may be loaded into RAM from a harddrive when a triggering event occurs. During execution of the softwaremodule, the processor may load some of the instructions into cache toincrease access speed. One or more cache lines may then be loaded into ageneral register file for execution by the processor. When referring tothe functionality of a software module, it will be understood that suchfunctionality is implemented by the processor when executinginstructions from that software module.

The following claims are not intended to be limited to the embodimentsshown herein, but are to be accorded the full scope consistent with thelanguage of the claims. Within a claim, reference to an element in thesingular is not intended to mean “one and only one” unless specificallyso stated, but rather “one or more.” Unless specifically statedotherwise, the term “some” refers to one or more. No claim element is tobe construed under the provisions of 35 U.S.C. § 112(f) unless theelement is expressly recited using the phrase “means for” or, in thecase of a method claim, the element is recited using the phrase “stepfor.” All structural and functional equivalents to the elements of thevarious embodiments described throughout this disclosure that are knownor later come to be known to those of ordinary skill in the art areexpressly incorporated herein by reference and are intended to beencompassed by the claims. Moreover, nothing disclosed herein isintended to be dedicated to the public regardless of whether suchdisclosure is explicitly recited in the claims.

What is claimed is:
 1. A method for performance testing computingservices, comprising: receiving a request to test application code, therequest including information identifying an amount of load generated bythe request in testing the application code; calculating one or moreminimal sets of resource pod configurations for the request based on theamount of load generated by the request; instantiating a plurality ofinstances implementing test code for the request using the one or moreminimal sets of resource pod configurations; executing the test codeusing the plurality of instances; retrieving test results generated byeach of the plurality of instances; generating a consolidated testresult by merging the test results generated by each of the plurality ofinstances, the consolidated test result including information about asuccess of executing the test code; and deploying the application codebased on determining that the consolidated test results meet a thresholdfor the application code.
 2. The method of claim 1, wherein the minimalsets of resource pod configurations for the request are calculated basedon a maximum amount of load that can be generated by each of a pluralityof resource pod configurations and the amount of load indicated in therequest, one or more minimal sets of resource pods needed to generatethe amount of load indicated in the request, the minimal set of resourcepods including a minimal set of one or more virtual compute instances(VCIs), the minimal set of one or more VCIs being selected using aresource minimization strategy that selects a lowest cost configurationof VCIs from a plurality of configurations, the plurality ofconfigurations including configurations selected from a plurality ofVCIs having a plurality of different supported transactions per second,each respective resource pod configuration being associated with arespective pod-level maximum load and includes one or more VCIs, andeach respective VCI being associated with a respectivecompute-instance-level maximum load.
 3. The method of claim 1, whereinthe consolidated test is generated by merging the test results generatedby each of the plurality of instances, the consolidated test resultincluding information about a total number of tests executed against theapplication code, a total number of tests that were executedsuccessfully by the application code, or a percentage of successfullyexecuted tests.
 4. The method of claim 3, wherein the test resultscomprise, for each instance of the plurality of instances, a number oftests executed against the application code, a responsiveness data, orcorrectness data.
 5. The method of claim 3, further comprising:committing the consolidated test result to a test report repository. 6.The method of claim 3, wherein the application code is deployed based ondetermining that the consolidated test results meet a thresholdperformance level comprising at least one of a threshold averageresponse time and a threshold response correctness level
 7. The methodof claim 1, wherein each respective configuration of the one or moreminimal sets of resource pod configurations specifies an amount ofmemory and an amount of processing power corresponding to the maximumamount of load the respective configuration is capable of generatingover a time period, a number of read requests to be executed over thetime period, and a number of write requests to be executed over the timeperiod.
 8. A system, comprising: a memory comprising executableinstructions; and a processor configured to execute the executableinstructions and cause the system to perform a method comprising:receiving a request to test application code, the request includinginformation identifying an amount of load generated by the request intesting the application code; calculating one or more minimal sets ofresource pod configurations for the request based on the amount of loadgenerated by the request; instantiating a plurality of instancesimplementing test code for the request using the one or more minimalsets of resource pod configurations; executing the test code using theplurality of instances; retrieving test results generated by each of theplurality of instances; generating a consolidated test result by mergingthe test results generated by each of the plurality of instances, theconsolidated test result including information about a success ofexecuting the test code; and deploying the application code based ondetermining that the consolidated test results meet a threshold for theapplication code.
 9. The method of claim 8, wherein the minimal sets ofresource pod configurations for the request are calculated based on amaximum amount of load that can be generated by each of a plurality ofresource pod configurations and the amount of load indicated in therequest, one or more minimal sets of resource pods needed to generatethe amount of load indicated in the request, the minimal set of resourcepods including a minimal set of one or more virtual compute instances(VCIs), the minimal set of one or more VCIs being selected using aresource minimization strategy that selects a lowest cost configurationof VCIs from a plurality of configurations, the plurality ofconfigurations including configurations selected from a plurality ofVCIs having a plurality of different supported transactions per second,each respective resource pod configuration being associated with arespective pod-level maximum load and includes one or more VCIs, andeach respective VCI being associated with a respectivecompute-instance-level maximum load.
 10. The method of claim 8, whereinthe consolidated test is generated by merging the test results generatedby each of the plurality of instances, the consolidated test resultincluding information about a total number of tests executed against theapplication code, a total number of tests that were executedsuccessfully by the application code, or a percentage of successfullyexecuted tests.
 11. The method of claim 10, wherein the test resultscomprise, for each instance of the plurality of instances, a number oftests executed against the application code, a responsiveness data, orcorrectness data.
 12. The method of claim 10, further comprising:committing the consolidated test result to a test report repository. 13.The method of claim 10, wherein the application code is deployed basedon determining that the consolidated test results meet a thresholdperformance level comprising at least one of a threshold averageresponse time and a threshold response correctness level
 14. The methodof claim 8, wherein each respective configuration of the one or moreminimal sets of resource pod configurations specifies an amount ofmemory and an amount of processing power corresponding to the maximumamount of load the respective configuration is capable of generatingover a time period, a number of read requests to be executed over thetime period, and a number of write requests to be executed over the timeperiod.
 15. A non-transitory computer readable medium storinginstruction executable by a processor to perform a method forperformance testing computing services, the method comprising: receivinga request to test application code, the request including informationidentifying an amount of load generated by the request in testing theapplication code; calculating one or more minimal sets of resource podconfigurations for the request based on the amount of load generated bythe request; instantiating a plurality of instances implementing testcode for the request using the one or more minimal sets of resource podconfigurations; executing the test code using the plurality ofinstances; retrieving test results generated by each of the plurality ofinstances; generating a consolidated test result by merging the testresults generated by each of the plurality of instances, theconsolidated test result including information about a success ofexecuting the test code; and deploying the application code based ondetermining that the consolidated test results meet a threshold for theapplication code.
 16. The method of claim 15, wherein the minimal setsof resource pod configurations for the request are calculated based on amaximum amount of load that can be generated by each of a plurality ofresource pod configurations and the amount of load indicated in therequest, one or more minimal sets of resource pods needed to generatethe amount of load indicated in the request, the minimal set of resourcepods including a minimal set of one or more virtual compute instances(VCIs), the minimal set of one or more VCIs being selected using aresource minimization strategy that selects a lowest cost configurationof VCIs from a plurality of configurations, the plurality ofconfigurations including configurations selected from a plurality ofVCIs having a plurality of different supported transactions per second,each respective resource pod configuration being associated with arespective pod-level maximum load and includes one or more VCIs, andeach respective VCI being associated with a respectivecompute-instance-level maximum load.
 17. The method of claim 15, whereinthe consolidated test is generated by merging the test results generatedby each of the plurality of instances, the consolidated test resultincluding information about a total number of tests executed against theapplication code, a total number of tests that were executedsuccessfully by the application code, or a percentage of successfullyexecuted tests and the application code is deployed based on determiningthat the consolidated test results meet a threshold performance levelcomprising at least one of a threshold average response time and athreshold response correctness level
 18. The method of claim 17, furthercomprising: committing the consolidated test result to a test reportrepository.
 19. The method of claim 17, wherein the test resultscomprise, for each instance of the plurality of instances, a number oftests executed against the application code, a responsiveness data, orcorrectness data.
 20. The method of claim 15, wherein each respectiveconfiguration of the one or more minimal sets of resource podconfigurations specifies an amount of memory and an amount of processingpower corresponding to the maximum amount of load the respectiveconfiguration is capable of generating over a time period, a number ofread requests to be executed over the time period, and a number of writerequests to be executed over the time period.